Remotely collaborative vehicle systems are becoming increasingly valuable due to their ability to engage in missions ranging from reconnaissance and intelligence-gathering to attack missions. One architecture often used to control remote vehicle systems involves streaming live video from the vehicle to an operator platform. The operator views the video feed and sends commands back to the vehicle based on the viewed video feed. Although such architecture ideally allows a remote operator to view the actual environment in which the vehicle is operating in real time, this architecture often has unmanageable latency issues and requires a significant amount of bandwidth such that real time operation is significantly hampered.
Latency is a time delay, such as the time that it takes a vehicle to capture, encode and send a video stream from the vehicle to the remote operator, plus the time that it takes for the operator to provide a control input, as well as the time it takes for the vehicle to receive and respond to the control input from the remote operator. Latency can produce undesirable effects, such as causing a vehicle to divert from a desired trajectory. For example, during a surveillance mission, if a vehicle sends a video feed with a delay of five seconds, when it is two seconds away from a desired surveillance point it will be too late for the operator to observe the latent video and provide a control input to be executed when the vehicle passes over the surveillance point. As a result, the operator will continuously be trying to compensate for the delay rather than operating the vehicle real time.
Another example of the impact of latency is when a vehicle measures its location and orientation, such as with a Global Positioning System (GPS) augmented with an Inertial Navigation Unit (INU). The kinematic status data, such as the vehicle's location, orientation, velocities, and accelerations are valid for the precise moment when they were obtained, but by the time that data is transmitted and observed by an operator the momentum of the vehicle will have carried it beyond the reported location. So, the operator will provide control inputs based on old kinematic data, not necessarily based on the state of the vehicle at the precise moment the control inputs were generated. This can cause undesirable effects such as pilot induced oscillation. Kinematic data latency can also hamper collaboration between the vehicles. For example, in a swarm of vehicles flying in close formation, the vehicles may exchange kinematic data with each other so that their control systems may continually adjust to maintain the desired formation. However, if there is latency associated with the kinematic data each vehicle's control systems will be basing their calculations on old kinematic data, not necessarily on data representing the status of other vehicles at the moment the calculations are being performed. Impacts of kinematic data latency could include needing to space the vehicles further from each other in the formation than would be desired to compensate for the effects of kinematic data latency or risk collisions between the vehicles or with objects in the environment.
In addition to latency, video streaming architectures also often require a significant amount of bandwidth. A high quality video stream could require one megabit per second or more of bandwidth. Such bandwidth is often taxing on the communication links between vehicles and remote operators. Multiple unmanned vehicle systems may operate collaboratively, such as in a swarm. Multiple video streams can increase the amount of bandwidth required to an unmanageable size.
An alternative architecture for remotely controlling vehicle systems involves compressing the video streams to reduce the bandwidth requirements. Although compressing the video streams advantageously reduces the bandwidth requirements, the compression and decompression overhead for such architecture may add several seconds or more to the latency. This added latency may make it difficult to remotely control the vehicle.
Another alternative architecture for remotely controlling vehicle systems involves tethering the vehicle to the remote operator. Tethering the vehicle advantageously provides a direct communication link between the vehicle and the operator and can provide sufficient bandwidth with minimal latency to enable effective operation of the tethered vehicle. However, in such an architecture, the mission of the tethered vehicle is limited to the length of the tether which greatly limits the types of missions that the vehicle is able to perform, as well as limits the ability of vehicles and operators to collaborate with others.